UMass at TDT 2004

Topic Detection classifies stories into different topics, but HTD requires more than that. Is there any other entities between a story and a topic? [10] views a topic as a structure of inter-related events, which gives us a good hint for this new task. Experiments in [10] show that time locality is a very useful attribute in event organization, and it can also help to solve the complexity problem in TDT2004. The TDT-5 collection contains 407,503 stories in three different languages, and the running time for traditional clustering algorithms, which take , is not acceptable for such a huge collection. Since we know that stories in the same event tend to be close in time, we only need to compare a story to its “local” stories instead of the whole collection. The algorithm we use has two steps, bounded 1-NN for event formation and bounded agglomerative clustering for building the hierarchy. In the first step, all stories in the same original language and from the same source are taken out and time ordered. Stories are processed one by one and each incoming story is compared to a certain number of stories before it. This number is approximately the number of stories in a token file and the value is 100 for the baseline run. If the similarity (cosine similarity of tf-idf term vectors) of the current story and the most similar previous story is larger than a given threshold (0.3 in the baseline run), the current story will be assigned to the event that the most similar previous